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First Soecial Report 


On 23 September 2004 the Science and Technology Committee published its Eleventh 
Report of Session 2003-04, Research Assessment Exercise: a re-assessment. On 23 
November 2004 the Committee received a memorandum from the Government which 
contained a response to the Report. The memorandum is published without comment as 
an appendix to this Report. 


Government response 


1. The Funding Bodies should have looked at the quality of the arguments set out in the 
responses to their consultation and not just the numbers. The move away from the 
“one size fits all” approach advocated by Sir Gareth Roberts is an important principle 
which should have been adopted. We consider the Funding Bodies to be unjustifiably 
conservative in their proposals. We do not see it as HEFCE's role to protect the 
sensitivities of universities. (Paragraph 16) 


In undertaking and considering the Roberts’ Review, the Funding Bodies engaged the 
sector in two rounds of extensive consultation. The results of these consultation exercises 
were analysed both quantitatively and qualitatively. A persuasive case was made against the 
multi-track assessment process, which was thought by many respondents to encourage 
unhelpful stratification of the sector. In developing the processes for RAE 2008, the 
Funding Bodies have taken into consideration the responses to the consultation on Sir 
Gareth Roberts’ proposals, and incorporated many of the key features into the assessment 
process. Ultimately the Funding Bodies are mindful that the RAE needs to carry credibility 
both internationally and nationally. This could not be achieved without substantial buy-in 
from the HE sector to the revised proposals. 


2. We believe that the panel/sub-panel structure is a positive step in improving 
consistency and improving the treatment of interdisciplinary research. We recommend 
that the Funding Councils seriously consider the establishment of panel moderators. 
(Paragraph 26) 


We welcome the recognition that the two-tier panel structure introduced for RAE 2008 is a 
positive step towards improving consistency of assessment and the treatment of 
interdisciplinary research. Ensuring consistency of assessment is a major function of the 
new Main panels chaired by eminent individuals. We believe that the panel structure— 
with sub-panel chairs sitting on the Main Panels, Main Panels working across a number of 
disciplines, and Panel Secretaries appointed to work across a Main Panel—meets the spirit 
of this recommendation. 


3. We welcome proposals to strengthen the use of overseas panel members. It should 
form part of a wider exercise to benchmark UK research. (Paragraph 32) 


We agree that benchmarking UK research is an important task. The Funding Bodies are 
working with the OST and others to consider other possible benchmarking tools and 
approaches. 


4. The panels and sub-panels need to be properly resourced. Overstretching staff and 
panel members could lead to panels reviewing individuals selectively and coming up 
with a biased or wrong conclusion on quality. Under-resourcing is an affront to the 
researchers and institutions who have gone to the trouble of putting the submissions 
together. (Paragraph 33) 


We agree the need for a properly resourced RAE. The Funding Bodies have taken this into 
account in the establishment of the core RAE Team; through the revisions to the 
appointment process for Panel Secretaries, who in 2008 will be seconded full-time from 
HEIs; as well as with the recruitment of Assistant Panel Secretaries to provide appropriate 
administrative cover for the exercise. On 28 September 2004, the four UK higher education 
Funding Bodies announced the appointment of 15 distinguished academics to chair the 
main panels for RAE 2008. The 15 main panels are part of a new two-tier panel structure 
for the 2008 exercise. 


5. It is clear that the workload of panels is excessive but it is less clear how it can be 
reduced while all higher education institutions continue to enter the RAE. An excellent 
opportunity to reduce the burden on panels has been missed by the Funding Bodies in 
their decision not to support different assessment routes. In particular our proposal to 
exempt top-ranking departments would reduce the number of submissions that each 
panel needed to consider and enable them to give closer consideration to submissions. 
(Paragraph 34) 


Whilst we acknowledge that the workload for the panel members is burdensome, the 
Funding Bodies’ consultation with the sector on the desirability of the “three track” 
assessment process had a mixed response. There was a substantial majority opposed to the 
idea, which runs counter to the desire for criterion referencing. Furthermore, in detailed 
consideration of the next process, the Funding Bodies were firmly persuaded that a 
competition based on subject review cannot exclude entire institutions. The two-tier panel 
system and the use of specialist advisers will significantly assist the assessment process. 


6. We conclude that the definition of research employed in 2001 is broadly adequate. It 
is important that the panels give equal weight to pure and applied research and that 
higher education institutions perceive this to be the case. HEFCE should ensure that it 
is understood by everyone. (Paragraph 41) 


We welcome this recommendation and agree that it is important that the definition is 
properly applied in the assessment of all forms of research. The revised framework is 
designed to ensure that applied research, joint submissions and interdisciplinary research 
are not disadvantaged. Panel membership will include people with experience of 
commissioning and using research, including those in industry, commerce and the public 
sector. The Funding Bodies are working with panel members, subject communities and 
institutions to ensure that this is understood. 


7. It is not acceptable for peer review panels to rely on the place of publication as a 
guarantee of quality. We recommend that HEFCE instruct panels to desist from this 
practice for RAE 2008 and ensure that panels are sufficiently large and well staffed to 
make informed judgements of the quality of the submissions. (Paragraph 42) 


The Funding Bodies will seek to ensure that the panels being established for the 2008 RAE 
are configured in such a way that they have within them, and through their special 
advisors, the competences they need to support informed judgement. Panels will not be 
allowed to make purely mechanistic judgments of quality. 


8. The RAE should recognise that excellent research may not be internationally 
significant but it may transform the fortunes of a local business or the provision of 
public services. We recommend that quality criteria concentrate more on the impact of 
research rather than the place where it has been published. (Paragraph 43) 


In determining the grade quality descriptors, the panels will take account of the fact that 
research of the highest quality may not always have an international dimension. 


9. We recommend that, once formed, panels publish rapidly clear guidance on how 
they plan to use metrics to inform their appraisals. This should be considered a 
priority. (Paragraph 46) 


There is widespread support in the higher education sector for an assessment process 
which is underpinned by peer review. The current RAE 2008 framework already allows for 
differential use of metrics in different disciplines; but this would be a matter for each 
subject panel to consider and consult with its subject community about. Notwithstanding 
this, the Funding Bodies are investigating how metrics may be used in future to support or 
challenge the decisions of panels. In September HEFCE jointly sponsored with OST an 
international workshop on the use of metrics to examine these issues. 


10. The introduction of a quality profile is a significant step forward and, if associated 
with an equitable funding formula, could eliminate many of the iniquities of the 
previous grading system. (Paragraph 49) 


We welcome this recognition that the quality profile is a potentially useful measure to 
reduce games playing and the difficulties of “cliff edges” in grading structure. We see this as 
one of the major reforms enabling the Government with Funding Bodies to assess and 
reward areas of research excellence wherever they might be found. 


11. We welcome HEFCE's acceptance that the tactics employed by universities to 
improve their RAE grade are not all part of a legitimate research strategy and 
recommend that it publishes analyses of the strategies being employed by institutions 
and provide guidelines on what it considers to be acceptable practices. (Paragraph 52) 


The Funding Bodies recognise that the RAE is as susceptible to “games playing” by those 
who participate in it as any other competition, and have taken steps to limit the degree of 
advantage which can be gained from such tactical activity. As in previous exercises, the 
Panels will consider institutional research strategies (both retrospective and forward- 
looking) in their assessment process. There will be Equal Opportunity training and 
guidance for panel members. Submitting institutions will also have to declare that they 
have Equal Opportunity policies in place, which have been operated in relation to the RAE 
submissions process. 


12. It seems likely that the media will defy the Funding Bodies’ best intentions and 
distil the quality profile down to a single figure, thus encouraging the exclusion of the 
least productive researchers if this is permitted. (Paragraph 54) 


The Funding Bodies are committed to publishing the results as “quality profiles” and will 
attempt to explain the procedures for collecting and assembling those profiles. Although it 
is true that single “league tables” will oversimplify this complex data, it is not possible to 
pre-judge how the media will interpret the results of the RAE. 


13. We think that greater transparency about the RAE, including the public disclosure 
of included academics, would have many benefits. It is important to highlight the 
valuable work done by excellent administrators and by lecturers who invest time and 
intellect in their teaching. We feel that greater clarity about the role of these academics 
could increase the value attached to their work. We recommend that as a condition of 
their block grant, higher education institutions publish an annual staff audit, 
describing the contributions of all members of academic staff to research, teaching, 
administrative and other functions. (Paragraph 55) 


14. In our earlier Report we reported that women academics were more likely to take 
on more of the teaching and pastoral functions within departments. It is our view that 
the issue to be resolved here is the status accorded to academics who take on these non- 
research but nonetheless essential roles. (Paragraph 56) 


(13,14) The Funding Bodies recognise the important work undertaken by all those 
involved in the Higher Education system, whether researchers, educators or 
administrators. The creation of the UK-wide HE Academy to focus specifically on the 
learning experience, including standards and development, enhancement, and research 
will further support the Funding Bodies’ individual initiatives. The review of research 
priorities, and the way that institutions plan, develop and use all their staff are matters for 
institutions to determine as part of the management of their organisation. 


However, it is not the place of the RAE to make judgments on the quality of either teaching 
or administrative work. Teaching quality is already assessed and reported by other means. 
An additional survey of the non-research contributions of all staff would in our view be an 
unwelcome extra reporting burden on the sector, and would add little value to the 
judgments formed either through the RAE or other assessment and monitoring processes. 


The Research Assessment Exercise makes no judgment about the quality of non-research 
activity undertaken, nor the status of those who undertake it, whether male or female. 
HEFCE has undertaken its own research into the issue of female participation in RAE 
2001, and will be discussing this issue with, amongst others, the AUT in the near future. 


15. HEFCE assures us that panel members, secretaries and RAE team staff will be 
bound by a duty of confidentiality. We anticipate that this will be challenged in the 
courts in a bid to reveal publicly the judgements made about the performance of 
researchers. We recommend that such a move should be pre-empted and that the 
grades awarded to individual researchers should be made public. This would bring 
welcome transparency to the process. (Paragraph 57) 


All those involved in the assessment process will be bound by a confidentiality agreement. 
Furthermore, it is very important that all those in the assessment process understand that 
the assessment is not an assessment of individuals, but rather it is an assessment of the 
quality of research undertaken by an institution in a submission to a unit of assessment. 
We will be discussing this with Panel members to ensure that they do not operate at the 
level of grading individual researchers. 


16. Although Sir Gareth's proposals for the assessment of “research competence”, if 
implemented, would place a heavier burden on the RAE, we believe that the need to 
promote good practice is so important that it should be used alongside other incentives 
to promote good practice in higher education research. (Paragraph 61) 


As noted in response to recommendation 11 above, the assessment of research strategies is 
an important part of the assessment process in the RAE. Furthermore, the Funding Bodies 
are introducing other measures of research competence (eg through introducing minimum 
standards for Postgraduate Research Degree programmes and Human Resource 
Strategies), which should help to ensure good practice in research. The RAE process itself 
explicitly recognises that institutions should have appropriate equal opportunities policies 
in place, paying particular attention to gender, ethnicity, disability and experience issues. 


17. We are disappointed that the Funding Bodies have rejected any form of mid-point 
monitoring. The RAE is designed to fund research excellence selectively and this 
funding should therefore reflect a department's current, and not only past, capabilities. 
(Paragraph 62) 


In the Funding Bodies’ consultation on the Robert’s Review, 80% of respondents were 
against mid-point monitoring. This was seen to place a much greater burden on 
institutions already concerned by the level of work required to submit to the RAE. 
Institutions themselves have the opportunity to distribute funds according to current 
capabilities and needs, as QR is made as a block grant to institutions, not to departments. 


18. The figures provided by the Funding Councils of the cost of the RAE to institutions 
do not appear to be excessive. The fact remains that this burden is resented by 
universities. The Funding Bodies should be sensitive to this feeling when developing 
their plans for 2008. (Paragraph 66) 


We welcome the acknowledgment that the overall cost of the RAE is not excessive. The 
Funding Bodies endeavour to keep the burden on institutions to the minimum required to 
produce robust results upon which to make important funding decisions. 


19. The Funding Bodies’ proposals have addressed positively many of our concerns 
about the RAE mechanism and HEFCE has adopted a more open-minded and 
constructive approach to its reform, which is a welcome change. A more radical 
approach, employing a range of metrics to reduce the bureaucratic burden on 
universities is still needed. We accept that their application will be a complex and time- 
consuming task for RAE and the Funding Bodies but we believe that the administrative 
burden should fall here rather than on the universities. (Paragraph 67) 


20. We conclude that a range of measures could be used to replace the peer review 
process in some subject areas, such as the physical sciences. There are strong reasons to 


believe that they could be as reliable as the current system while being more cost 
effective and imposing less of a burden on institutions and panel members. We 
recommend that the Funding Bodies commission an external study to consider 
options. (Paragraph 74) 


(19, 20) We welcome the Committee’s acknowledgement of the positive steps the Funding 
Bodies have taken to reform the RAE mechanism. The Science and Innovation Investment 
Framework 2004-2014 sets out the Government’s commitment to develop metrics to 
shadow RAE 2008 to provide robust information to benchmark the outcomes of the RAE 
exercise (para 3.48). This will ensure that the metrics collected as part of the next 
assessment exercise will also be available to undertake an exercise shadowing the 2008 RAE 
itself. The aim will be to provide a benchmark on the information value of the metrics as 
compared to the outcomes of the full peer review process. DfES and OST are working 
closely with HEFCE and other partners (including international expertise) to explore the 
use of a range of metrics, and the Funding Bodies and key stakeholders are looking beyond 
the 2008 RAE to see how metrics can be most sensibly deployed to reduce the burden on 
institutions without compromising the credibility and reliability of the exercise. 


21. We accept that there are practical difficulties in delaying the next RAE and 
recommend that the RAE continue as proposed in 2008 but that the Funding Bodies 
draw up a clear timetable for the development of alternative models of research 
assessment. (Paragraph 77) 


As stated at the beginning of this response, the Government and the Funding Bodies 
welcome the Committee’s recommendation that the RAE should go ahead as planned in 
2008. All aspects of the RAE process will be kept under careful review and inform decisions 
about research assessment in the longer term. 


22. Departments need to know how to play the RAE game, yet HEFCE is asking them to 
do it blindfolded. HEFCE should draw up guidance to universities on how the quality 
profile will be used to calculate the funding. We appreciate that there are a number of 
variables that cannot be known in advance of the RAE but HEFCE should have the 
capability to produce estimates which would enable it to provide indications about the 
level of funding provided to each band of the profile. It should do this without delay. 
(Paragraph 79) 


It is important to separate the need for transparency and support for institutions and 
departments which need to know how to follow the rules, from unhelpful games playing. 
The priority is to fund the best research, but we are mindful of Sir Gareth Roberts’ 
recommendation about the steep cliff edge. As his report says, there are too many 
unknown variables - the gradient of each quality profile; the numbers of departments 
submitted; the numbers of staff submitted and the exact level of funding from Government 
— to make it virtually impossible to make any definitive statements on funding prior to the 
RAE results being known. The general thrust of the report welcomes attempts by the 
Funding Bodies to minimise games playing (para 80). The Government will work with 
HEFCE on funding to ensure that the sector is clear about our overall policy objectives, and 
it will consult with HEFCE about funding levels within our overall policy of excellence. 


23. It is not clear to us why HEFCE has deemed it necessary to further increase the level 
of selectivity of QR funding. We regret that it will intensify many of the problems 
caused by the RAE and the funding decisions based on it. (Paragraph 81) 


The Government's policy is to target resources selectively on the best quality research to 
sustain world class status in the long run. Within the context of finite funds, it is important 
that the Funding Bodies should continue to fund the highest quality research (as judged by 
the RAE) at a sustainable level. The selectivity ratio will therefore vary according to both 
the overall performance by the sector in the 2001 RAE and the funding made available by 
Government. A less selective ratio for funding would have to be at the expense of reducing 
funding for 5 and 5* departments. 


24. We welcome HEFCE's capability funding as a means of building research capability 
and promoting dynamism in the research base. We are concerned, however, that it is 
too restrictive. We believe that all departments should be eligible and grants should be 
awarded on the strengths of their research and investment strategies. (Paragraph 84) 


The Government believes that HEFCE’s present policy to support research in emerging 
areas is appropriate. HEFCE’s research capability fund is a recognition that certain subjects 
have a less well established research base. It is targeted on those subjects and on 
departments in those subjects that have demonstrated a reasonably strong base of existing 
achievement on which to build. HEFCE has reviewed its capability funding and has agreed 
to work with Research Councils and other partners to identify and jointly fund research in 
sub-disciplines (for example chemical engineering or nuclear physics) where the UK’s 
capacity is threatened. 


25. We are pleased that Sir Howard Newby now recognises that a policy of highly 
selective research funding, based on the RAE, has had an effect on the viability of 
university departments in core subjects. The RAE does not take place in a vacuum and 
further changes are also now taking place in higher education following the 2003 White 
Paper and the introduction of variable tuition fees. It is too early to say what the precise 
impact of those changes will be. Concerns expressed so far, however, suggest that 
variable fees may also lead to closures of further university departments, quite possibly 
in the physical sciences. The operation of the RAE and variable fees may, therefore be 
mutually self re-enforcing and HEFCE should remain vigilant in these respects. 
(Paragraph 90) 


The factors affecting departmental closures are complex. The principal driver for closures 
is a lack of student demand in certain subjects. In some cases, it may be part of the 
institution’s own financial management and institutional strategies, which may be 
accentuated by poor research performance. The ten year investment framework asks 
HEFCE to work with a range of stakeholders to co-ordinate national effort to address these 
subjects. 


26. The provisions for HEFCE to delay closure or offer funding to struggling 
departments have been criticised for threatening the autonomy of universities but this 
encroachment on their independence is a price worth paying for the preservation of 
core disciplines on a national basis. We accept that these powers should be used with 
restraint but this is an important shift in policy we welcome. (Paragraph 91) 


HEFCE has welcomed the Government's request for the Council to explore the proposal 
that institutions should give notice to HEFCE of departmental closures. The issue is not 
about stopping autonomous individual institutions making sensible business decisions. It 
is about the sum of individual decisions having a negative impact on access to certain 
subject provision in regions and sub-regions. 


27. The Government says it will assess the “trajectory” towards the full economic cost 
model in 2006 in time for the next Spending Review. We hope that it will use the 
opportunity to review whether the model is a viable one and whether the aim of 
rebalancing the dual support system could be achieved by a straightforward increase in 
the research funds available to the Funding Bodies. (Paragraph 94) 


The Government has stated its commitment to Dual Support and to continue to increase 
funding for both sides of the Dual Support system. This has been welcomed by the 
Funding Bodies and the sector. It is recognised that the Funding Bodies’ side of dual 
support has not kept pace with the other side. The Government's intention is not to 
increase either leg of the Dual Support system at the expense of the other, but to achieve 
balance on both sides of the system against a rising overall investment. 


We support the principle that institutions should understand the costs of their various 
activities and that they should aim to operate on a sustainable basis. HEFCE’s Financial 
Memorandum with institutions requires them to recover the Full Economic Costs across 
the full range of their activities. Not to do so would continue to erode the HE asset base 
with long term damage to the higher education system. OST has worked closely with the 
sector to ensure the modelling base for Full Economic Costs (FEC) is based on robust data. 
The percentage of FEC and the timetable for reaching 100% FEC are due to be announced 
in early December. 


28. We would like to see diversity in higher education research funding but it is hard to 
see how this can be achieved while the RAE dominates the funding landscape. We have 
concluded that new incentives for all areas of universities’ work are needed. Quality 
assessment for teaching has proved problematic and unpopular. The Government 
should consider more radical solutions, perhaps awarding teaching funds on the basis 
of outputs rather than inputs as has been the case. The “third leg” funds for knowledge 
transfer have grown in recent years but it is nor clear whether they are yet sufficient to 
act as an adequate counterbalance to RAE-based funding. We conclude that a greater 
diversity of funding streams would act as a counterbalance to the RAE. The proposed 
European Research Council could contribute, as would the greater availability of 
research funds from other Government Departments. (Paragraph 96) 


The ten year investment framework (in paragraph 3.4) highlights one of the strengths of 
the UK university system as its ability to secure a wide range of current and capital funding 
for research from a variety of public, private and other sources, ie Other Government 
Departments, UK based charities, UK industry, EU bodies and others. In the HE White 
Paper, the Government recognised the great deal of diversity that exists within the sector. 
We believe this needs to be further acknowledged and supported by a funding regime 
which enables each institution to choose its mission and the funding streams necessary to 
support it; and to make sure that our system recognises, celebrates and encourages 
different missions properly. Knowledge transfer is of particular importance. As noted in 


the ten year investment framework, the Government's aim is to create a funding regime 
which promotes and rewards high quality knowledge transfer. As part of the 2004 
Spending Review settlement, funding available for the Higher Education Innovation Fund 
(HEIF) will increase to £110 m a year by 2007-2008. The Government will also move 
towards a predictable funding allocation to HEIs on a national basis for knowledge 
transfer. This new allocation process will be introduced for a substantial part of the next 
round of HEIF. 
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